With the rapid growth of video content in recent years, there has been an increased demand for video-related applications in various industries, such as broadcasting, content generation, surveillance, and so forth. In some cases, these video-related applications rely on video analysis techniques that may include indexing, annotation and/or searching performed in relation to video streams and its context. Further, real time or near real time video analysis can be useful for enabling advanced applications, such as accelerated processing of large video archives, rapid provision of video content in the broadcasting industry, or the like. However, real time video analysis for typical indexing tasks, e.g., object detection, object recognition, object tracking, scene detection, action detection, pose estimation, etc., can be challenging, such as in the case of high video frame rates and/or high resolution video, which can increase the load on available computational resources. For example, conventional solutions for real time video analysis are not highly dynamic and are typically aimed at specific tasks. In addition, conventional solutions may not be extensible and may not be able to adapt dynamically during real time video analysis.